1. Field of the Invention
The present invention relates, generally, to the processing of change notifications when a change to a device has occurred on a loop, and in particular embodiments to the buffering of Registered State Change Notifications (RSCNs) to initiators to reduce the overhead required to process RSCNs.
2. Description of Related Art
Conventional and legacy storage systems utilize disk drives connected together in a loop, and implement a protocol such as a Fibre Channel Arbitrated Loop (FC_AL) protocol for communicating between devices connected to the loop. FIG. 1 illustrates an exemplary private loop 100 containing a number of devices 102. Note that a device, as referred to herein, includes, but is not limited to, disk drives, host bus adapters (HBAs) and other Fibre Channel (FC) devices. The private loop 100 may be referred to a Just a Bunch Of Disks (JBODs).
When a new device 104 is first connected to the private loop 100, all of the devices previously connected to the private loop 100 must be initialized using a Loop Initialization Primitive (LIP) cycle. As part of the LIP cycle, the new device 104 sends out a Loop Initialization Select Master (LISM) frame 106 to all devices in the loop 100, one by one, with the address (worldwide name) of the new device 104 initially stored into the LISM frame 106. As other devices in the loop receive the LISM frame 106, the address stored in the LISM frame 106 is checked. If a device has a greater worldwide name (lower number), it replaces the address in the LISM frame 106 with its own greater worldwide name. Eventually, the LISM frame 106 arrives back at the new device 104, containing the greatest worldwide name of any of the devices in the loop 100. The device with the greatest worldwide name (lowest number) is designated as the loop initialization master (LIM) of the initialization phase. The new LIM device sends ARB(f0) ordered sets around the loop to indicate that a master has been selected. From that point forward, the device designated as the LIM sends out future initialization frames to determine the addresses of all devices on the loop 100. Note that when one of the devices on the loop is an initiator such as an HBA (see reference character 108), the initiator is made aware of all of the devices on the loop 100 and their addresses as a result of the LIP cycle.
When a device is removed from the loop, the LIP cycle is repeated, and the process described above is performed to once again determine the addresses of all devices remaining on the loop.
As illustrated in the exemplary interconnection diagram of a storage system shown in FIG. 2, a non-blocking frame-based crossbar switch (e.g. frame-based switch 200) enable the interconnection of a large number of devices such as JBODs 202, 204, 206, and 208, and an initiator 212. The devices are connected to ports in the frame-based switch 200. Note that unlike the FC_AL private loop of FIG. 1, which utilizes an 8-bit Arbitrated Loop Protocol Address (ALPA) and has a 126 device limit, frame-based switches implement a fabric services protocol with a 24-bit address (which includes an 8-bit domain address and an 8-bit area address in addition to an 8-bit ALPA), which provides for a much higher device limit. As illustrated in FIG. 2, frame-based switches implementing the fabric services protocol support the interconnection of a private loop on each port.
In the configuration of FIG. 2, when a new device 210 is added to JBOD 202, the LIP cycle described above will be executed, and as a result all devices in JBOD 202 and the frame-based switch 200 will be provided with the address of not only the new device but all other devices in JBOD 202. However, because the initiator 212 is not connected directly to the JBOD 202 but is separated by the frame-based switch 200, the initiator 212 is not provided with the address and other information of the devices in JBOD 202.
FIG. 3 is an illustration of an exemplary frame-based switch 300 connected to an initiator 302 and various other JBODs 304. Each JBOD 304 may contain up to 126 devices, as limited by the 8-bit ALPA discussed above. The frame-based switch includes a router 310, a processor 306, a switch core 312, and ports 314. To provide the initiator 302 with information (i.e. address and other information) about the devices connected to the JBODs 304, a well-known protocol has been developed wherein the initiator 302 first sends a State Change Registration (SCR) frame 316 to a state change manager 308 (a.k.a. fabric controller) implemented in firmware by processor 306, indicating to the state change manager 308 that the initiator 302 wants to be notified when the devices in the JBODs 304 have changed. The SCR frame 316 includes a FC header 334 including the address of the initiator 302, an Extended Link Service (ELS) byte 326, an SCR byte 328 identifying the frame as an SCR frame, and an information field 330 which indicates whether the initiator 302 wants to be notified each time a device has changed, or each time a port has changed, or each time any device or port has changed. The state change manager 308 maintains a database 332 of all initiators that have registered to receive notifications of state changes (i.e. all initiators that have sent an SCR to the state change manager 308), and what types of notifications are to be sent to each initiator.
A name server 318 maintains a database 320 of information related to each of the devices attached to the frame-based switch 300, including their 24-bit address (described above), the world-wide name (WWN) of the device, and the FC-4 type (which indicates whether the device is an initiator, target, etc.). The name server 318 may also be implemented in firmware executed by the processor 306. The database 320 is needed because of the large number of devices potentially attached to the frame-based switch 300. Note that the database 320 does not include duplicate entries. When changes to a device (as identified by its 24-bit address) are made, the device sends a registration to the name server 318, and old information in the database 320 is immediately discarded and replaced by the new information. There are actually two phases to the registration. When a device is connected to a switch that provides fabric services (including the name server), it uses an FLOGI command to request a 24-bit address from the switch. When the switch receives the FLOGI ELS it registers the device in the name server database as it knows the 24-bit address and the WWN of the device. It does not know the FC-4 type yet. The next step the device will perform is logging into the name server via a PLOGI ELS and then performing a “register FC-4 types” command. This is command number 0-0217 in the name server. At this point the name server has the complete set of information about the device.
When changes are made to the name server database 320, change notifications 322 are sent to the state change manager 308. Change notifications are internal notifications inside the switch 500 comprised of data items and function calls. When a change notification 322 is received by the state change manager 308, the state change manager 308 determines from database 320 which initiators have registered to receive notifications of that particular state change, and then sends a Registered State Change Notification (RSCN) 324 to each initiator that had previously registered to receive that state change.
The RSCN 324 contains the 24-bit address of the device that changed, and an additional byte (eight bits) of information that specifies how much of the 24-bit address is valid. The possible values for that byte indicate that either: 1) the whole 24-bit address is valid, 2) the ALPA part of the address is to be ignored, or 3) the ALPA and area part of the address are to be ignored. Option 1) is used when a single device is added or deleted, and when the entire loop is plugged in. Option 2) is useful when the link between the loop and the fabric switch goes down, or when the loop is removed from the switch. An RSCN can be sent to indicate that all ALPAs with this domain and area have changed. In other words, regardless of whether a device was added or deleted, the RSCN 324 would contain the same 24-bit address for that device. Thus, after receiving an RSCN 324, an initiator must then query the name server 318 for additional details regarding the change. A protocol used within the name server 318 allows the initiator to make queries to the name server 318, such as whether a device exists at a particular address, the type of device that exists at the particular address, or other details about the device. These query commands are well-documented in the FC specifications. If the device is no longer found in the name server database 320, the initiator knows that the device has been removed from the loop. If the device is found in the name server database 320, then the initiator knows that the device was added, and can query the name server as described above to receive additional information about the added device.
FIG. 4 illustrates multiple frame-based switches 400 connected together in a single storage enclosure 402. The frame-based switches are connected together via a System Packet Interface (SPI) bus 404 daisy chained to SPI input and output ports on each frame-based switch 400. SPI is a multi-channel protocol time-division multiplexed (TDMed) over a single bus. The SPI bus 404 includes a control line, 16 data lines, and a clock. In the multi-switch environment of FIG. 4, each name server 418 maintains a database of information related to each of the devices connected to each frame-based switch 400. Each state change manager 408 in each frame-based switch 400 maintains a database of all local initiators 406 (i.e. initiators connected to that particular frame-based switch 400) that have registered to receive state changes (i.e. all local initiators that have sent an SCR to the state change manager within the frame-based switch to which the local initiator is connected).
Due the initialization procedure that is performed every time a device is added to or removed from a loop, when even a single device on a JBOD is added or removed, every device in that JBOD is initialized (even though it may receive the same address after initialization), and a large number of registrations will be sent to the name server database. As these changes are made to the name server database, a large number of change notifications are sent to the state change manager, and a large number of RSCNs are sent to the registered initiators, one for each changed device. For example, assume that a JBOD has 125 devices connected to it. When one of the devices is removed from the JBOD, a LIP cycle is performed, 125 delete registrations are sent to the name server, and 125 RSCNs are sent to the registered initiators. When the LIP cycle is completed for the remaining 124 devices, 124 add registrations are sent to the name server, and 124 RSCNs are sent to the registered initiators. When a new device is attached to the JBOD to replace the device that was removed, a LIP cycle is performed, 124 delete registrations are sent to the name server, and 124 RSCNs are sent to the registered initiators. When the LIP cycle is completed for the 125 devices, 125 add registrations are sent to the name server, and 125 RSCNs are sent to the registered initiators. Thus, the replacement of a single device into a JBOD can cause a large number of additions and deletions to be communicated to the name server, and a large number of RSCNs to be sent to the registered initiators. When an entire JBOD or all or part of the storage system that includes the frame-based switch and the JBODs is powered up, many devices have state changes, and there will be many changes to the name server database, many change notifications sent to the state change manager, and many RSCNs sent to the registered initiators, which increases traffic and reduces data throughput for the storage system. In addition, the response time (the time that a device is changed to the time that an initiator receives information about the change) is slowed down when numerous RSCNs are being processed.
Therefore, there is a need to process a large number of device changes in a storage system in an efficient manner to improve throughput and reduce traffic.